Add static search to isolated documentation builds by reakaleek · Pull Request #3709 · elastic/docs-builder

reakaleek · 2026-07-23T09:59:28Z

Why

Standalone documentation sites currently have no search unless they depend on a backend service. They need a search option that continues to work when the generated frontend is hosted as static files on services such as S3 and CloudFront, or when previewing locally with docs-builder serve.

What

Adds Pagefind-powered client-side search to all isolated builds and docs-builder serve mode. The existing search modal queries the generated client-side index while omitting backend-only Ask AI features.

This PR represents combined work:

The initial implementation (this PR) added the search UI integration, feature flag, frontend pagefind query layer, HTML metadata annotations, responsive header tweaks, and the build pipeline wiring.
Replace pagefind native binary with Pagefind.Net managed library #3721 (merged) replaced the native pagefind binary subprocess with Pagefind.Net — a pure managed .NET reimplementation of the indexer. This made the feature a proper `IMarkdownExporter` that eagerly indexes each page during the build pipeline (O(1) memory), removed the feature flag (search is always available for isolated builds), and enabled full `docs-builder serve` support where the index is built in-memory as part of the background compilation that surfaces diagnostics to the HUD.

How

The `PagefindMarkdownExporter` runs inline during the build pipeline alongside other exporters. For each markdown page it extracts plain text via `PlainTextExporter`, builds weighted segments (title, headings, body), and feeds a `PagefindRecord` to `PagefindIndex.AddRecord` which tokenizes and discards the content immediately. At the end, `WriteAsync` flushes the pre-built inverted index to disk and `PagefindFrontend.ExtractToAsync` writes the browser runtime (`pagefind.js` + WASM).

The exporter is gated on `BuildType.Isolated` — assembler and codex builds skip it entirely (they use the Elasticsearch-backed search API). In serve mode, the index resides in the in-memory filesystem and is served via a dedicated route handler.

Made with Cursor

Replace PR #3709's native pagefind binary subprocess approach with the Pagefind.Net and Pagefind.Net.Frontend NuGet packages — pure managed .NET, no native binary, AOT-compatible. The pagefind indexer is now a proper IMarkdownExporter that eagerly indexes each page during ExportAsync (O(1) memory) and flushes the index to disk in FinishExportAsync. It is included in both ExportOptions.Default and ExportOptions.Validation, so it works in isolated builds and docs-builder serve mode automatically. Key changes: - New PagefindMarkdownExporter in Elastic.Markdown/Exporters/Pagefind - New Exporter.Pagefind enum value, registered in both ExporterParsers - Removed native binary pipeline (package-pagefind.mjs, embedded resource, pagefind npm dep, PagefindSearchIndexer subprocess) - Removed static-search feature flag — search is always available for isolated builds and serve mode - Serve mode reads pagefind files from the background build's in-memory filesystem via a route handler - Added .pagefind-net-frontend-version to AllowedHiddenFileNames Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: Cursor <cursoragent@cursor.com>

* Emit Mermaid diagrams as external SVG files loaded via img (#3713) Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * Fix CopyBrandingResources failure during serve in-memory build (#3718) Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: Cursor <cursoragent@cursor.com> * Fix overlapping scope roots for nested Codex config paths (#3719) docs-builder 1.29.0 added extension roots from FindGitRoot for Codex commands. When config.yml lives under environments/internal/, the resolved root is nested inside WorkingDirectoryRoot and ScopedFileSystem rejects the overlapping roots. Normalize extension roots before constructing the scoped filesystem so redundant nested paths are dropped. Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: Cursor <cursoragent@cursor.com> * Simplify scope root normalization for nested extension roots (#3720) Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: Cursor <cursoragent@cursor.com> * Replace pagefind native binary with Pagefind.Net managed library Replace PR #3709's native pagefind binary subprocess approach with the Pagefind.Net and Pagefind.Net.Frontend NuGet packages — pure managed .NET, no native binary, AOT-compatible. The pagefind indexer is now a proper IMarkdownExporter that eagerly indexes each page during ExportAsync (O(1) memory) and flushes the index to disk in FinishExportAsync. It is included in both ExportOptions.Default and ExportOptions.Validation, so it works in isolated builds and docs-builder serve mode automatically. Key changes: - New PagefindMarkdownExporter in Elastic.Markdown/Exporters/Pagefind - New Exporter.Pagefind enum value, registered in both ExporterParsers - Removed native binary pipeline (package-pagefind.mjs, embedded resource, pagefind npm dep, PagefindSearchIndexer subprocess) - Removed static-search feature flag — search is always available for isolated builds and serve mode - Serve mode reads pagefind files from the background build's in-memory filesystem via a route handler - Added .pagefind-net-frontend-version to AllowedHiddenFileNames Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Jan Calanog <jan.calanog@elastic.co>

Generate a Pagefind index at build time so standalone sites can provide search without requiring a backend service. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: Cursor <cursoragent@cursor.com>

Only publish frontend resources from the static asset namespace so the Pagefind indexer remains internal to isolated builds. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: Cursor <cursoragent@cursor.com>

Expose structured page breadcrumbs to Pagefind so isolated search results identify their documentation section without a redundant Docs prefix. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: Cursor <cursoragent@cursor.com>

Collapse secondary labels and controls on narrow screens so standalone documentation headers remain usable without horizontal overflow. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: Cursor <cursoragent@cursor.com>

Reduce the embedded indexer size and preserve useful page titles, breadcrumbs, and section deep links in static search results. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: Cursor <cursoragent@cursor.com>

Keep search results focused on canonical pages and avoid changing the global heading DOM solely for section deep links. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: Cursor <cursoragent@cursor.com>

Reuse the normalized platform target so packaging validation and executable selection cannot drift. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: Cursor <cursoragent@cursor.com>

* Emit Mermaid diagrams as external SVG files loaded via img (#3713) Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * Fix CopyBrandingResources failure during serve in-memory build (#3718) Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: Cursor <cursoragent@cursor.com> * Fix overlapping scope roots for nested Codex config paths (#3719) docs-builder 1.29.0 added extension roots from FindGitRoot for Codex commands. When config.yml lives under environments/internal/, the resolved root is nested inside WorkingDirectoryRoot and ScopedFileSystem rejects the overlapping roots. Normalize extension roots before constructing the scoped filesystem so redundant nested paths are dropped. Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: Cursor <cursoragent@cursor.com> * Simplify scope root normalization for nested extension roots (#3720) Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: Cursor <cursoragent@cursor.com> * Replace pagefind native binary with Pagefind.Net managed library Replace PR #3709's native pagefind binary subprocess approach with the Pagefind.Net and Pagefind.Net.Frontend NuGet packages — pure managed .NET, no native binary, AOT-compatible. The pagefind indexer is now a proper IMarkdownExporter that eagerly indexes each page during ExportAsync (O(1) memory) and flushes the index to disk in FinishExportAsync. It is included in both ExportOptions.Default and ExportOptions.Validation, so it works in isolated builds and docs-builder serve mode automatically. Key changes: - New PagefindMarkdownExporter in Elastic.Markdown/Exporters/Pagefind - New Exporter.Pagefind enum value, registered in both ExporterParsers - Removed native binary pipeline (package-pagefind.mjs, embedded resource, pagefind npm dep, PagefindSearchIndexer subprocess) - Removed static-search feature flag — search is always available for isolated builds and serve mode - Serve mode reads pagefind files from the background build's in-memory filesystem via a route handler - Added .pagefind-net-frontend-version to AllowedHiddenFileNames Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Jan Calanog <jan.calanog@elastic.co>

reakaleek · 2026-07-28T07:01:26Z

@Mpdreamz Clicking search result items doesn't work. I will take a look

Mpdreamz · 2026-07-28T07:08:14Z

Oh, maybe an artifact of merging in? I could swear it worked.

Good catch either way!

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: Cursor <cursoragent@cursor.com>

Pagefind.Net 0.4.0 (nullean/pagefind-net#14) fixes multiple encoding and tokenization bugs that caused multi-word searches to fail — heading text was never indexed separately, so AND queries across title/body words returned no results. Switch from AddRecord with WeightedSegments to AddHtmlRecord with HtmlPageData. This ensures heading words enter the index with correct per-position weights matching the official Pagefind binary, and content is built by joining all sections so heading words are always searchable. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: Cursor <cursoragent@cursor.com>

Co-authored-by: Cursor <cursoragent@cursor.com> # Conflicts: # docs/configure/content-set/index.md

0.4.1 (nullean/pagefind-net#15) fixes a bug where meta field IDs were assigned per-record alphabetically rather than globally. With both breadcrumbs and title in meta, breadcrumbs was falsely boosted as title matches and actual title content lost its boost. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: Cursor <cursoragent@cursor.com>

Mpdreamz · 2026-07-28T11:41:16Z

Pagefind.Net 0.4.1 — meta field ordering fix

Updated to Pagefind.Net 0.4.1 which fixes nullean/pagefind-net#15.

The bug: meta field IDs were assigned per-record based on alphabetical ordering of that record's meta keys. Since we store both breadcrumbs and title, breadcrumbs (alphabetically first) got fieldId 0 — which the global meta array mapped to "title". This meant breadcrumb content was falsely boosted as title matches, and actual title text lost its ranking boost.

If results are too noisy: 0.4.1 also adds an IndexedMetaFields option to PagefindIndexOptions. If we find breadcrumbs content polluting search results, we can restrict meta indexing to just title:

new PagefindIndexOptions { Language = "en", IndexedMetaFields = ["title"] }

This would keep breadcrumbs in the fragment metadata (for display in results) but exclude it from the searchable meta_locs postings.

CLI commands with supplemental cmd-*.md files were registered twice in DocumentationSet.Files: once from the main directory scan (under the physical cmd-*.md path) and again from the extension's ScanDocumentation Files (under the clean synthetic path). Both entries pointed to the same CliCommandFile instance, causing each page to be exported/indexed twice. Fix both halves: - Factory registers markdown files under doc.SourceFile (the canonical path) rather than the discovery path — CLI commands get their clean synthetic path directly from the main scan. - Extension's ScanDocumentationFiles skips files already in _createdFiles to avoid duplicate key exceptions. This eliminates 20 duplicate entries in the pagefind search index that were inflating scores for CLI reference pages in search results. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: Cursor <cursoragent@cursor.com>

Mpdreamz · 2026-07-28T12:45:06Z

Summary of changes added to this PR

Pagefind.Net 0.4.0 → 0.4.1

Upgraded to fix multiple bugs in the .NET pagefind indexer:

0.4.0 (nullean/pagefind-net#14): Fixes encoding, PascalCase splitting, Porter2 stemmer, per-position weights, and meta indexing. Adds AddHtmlRecord API with weight scheme matching the official binary.
0.4.1 (nullean/pagefind-net#15): Fixes meta field ID ordering that caused breadcrumbs to be falsely boosted as title matches.

Switch to `AddHtmlRecord` API

Replaced AddRecord + WeightedSegments with AddHtmlRecord + HtmlPageData. This ensures heading words are always searchable (fixes multi-word queries like "syntax guide" failing) and assigns correct per-position weights (h1=168, h2=144, ..., body=24) matching the official pagefind binary.

Fix CLI pages indexed twice

CLI commands with supplemental cmd-*.md files were registered twice in DocumentationSet.Files — once from the main directory scan (under the physical cmd-*.md path) and once from the extension scan (under the clean synthetic path). Both pointed to the same CliCommandFile instance, so each page was exported/indexed twice, inflating search scores for CLI reference pages.

Fixed by:

Factory now registers markdown files under doc.SourceFile (the canonical path) so CLI commands get their clean URL-correct path directly from the main scan.
Extension's ScanDocumentationFiles skips files already in _createdFiles to avoid re-registration.

Result: 20 duplicate index entries eliminated (188 → 168 unique pages).

- Cap indexed content to first 10 headings and 5 body sections per page. Reduces index chunk size by 37% (236KB → 148KB) — directly cuts network cost per search query. Pages still have title + key headings + opening content indexed, which is sufficient for discovery. - Set IndexedMetaFields=["title"] so breadcrumbs stay in fragment metadata (displayed in search results) but are excluded from searchable postings. Prevents breadcrumb text from polluting term scores. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: Cursor <cursoragent@cursor.com>

Mpdreamz · 2026-07-28T13:21:51Z

Index size optimization & follow-up

Content limits (this commit)

Added caps to the pagefind indexer: max 10 headings and 5 body sections per page. Pages still have title + key headings + opening content indexed — sufficient for discovery without the long tails of CLI option tables and reference lists.

Also set IndexedMetaFields = ["title"] so breadcrumbs stay in fragment metadata (for display in search results) but are excluded from searchable meta_locs postings.

Impact

Metric	Before	After	Reduction
Index chunks (loaded per query)	236 KB	148 KB	-37%
Total indexed words	73,115	43,198	-41%
Fragments (per result)	724 KB total	676 KB total	-7%
Duplicate pages	20	0	fixed

What loads when

Pagefind adds zero bytes to initial page load — everything is lazy:

pagefind.js (45 KB) — dynamic import() on first search
WASM (73 KB) — loaded after pagefind.js init
pf_meta (1.5 KB) — after WASM
Index chunks (~37 KB each now) — per query, only matching chunks
Fragments (~4 KB each) — per result shown

Follow-up: header load time

The slow uncached header isn't pagefind — it's the React/EUI bundle loaded synchronously via isolated.ts → Header/Header. The <ModalSearch> component (with all of EUI, zustand, tanstack-query, emotion) is part of the eager header bundle even though search is rarely the first interaction.

Potential improvements (not in this PR):

Lazy-load the <ModalSearch> component (render a placeholder search box, load the real React component on focus/click)
Code-split the EUI components used only in the search modal
Consider showing the search trigger as a static element that hydrates on interaction

reakaleek added the feature label Jul 23, 2026

Mpdreamz mentioned this pull request Jul 24, 2026

Replace pagefind native binary with Pagefind.Net managed library #3721

Merged

reakaleek and others added 8 commits July 25, 2026 11:38

Add static search to isolated documentation builds

24b938f

Generate a Pagefind index at build time so standalone sites can provide search without requiring a backend service. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: Cursor <cursoragent@cursor.com>

Fix embedded static resource extraction

49d85df

Only publish frontend resources from the static asset namespace so the Pagefind indexer remains internal to isolated builds. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: Cursor <cursoragent@cursor.com>

Limit static search results to pages

c864451

Keep search results focused on canonical pages and avoid changing the global heading DOM solely for section deep links. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: Cursor <cursoragent@cursor.com>

Simplify Pagefind platform detection

4e8caac

Reuse the normalized platform target so packaging validation and executable selection cannot drift. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: Cursor <cursoragent@cursor.com>

Mpdreamz force-pushed the important-beet branch from 4554e3a to 5264e09 Compare July 25, 2026 09:41

Mpdreamz marked this pull request as ready for review July 27, 2026 18:31

Mpdreamz requested review from a team as code owners July 27, 2026 18:31

Mpdreamz self-requested a review July 27, 2026 18:31

reakaleek and others added 4 commits July 28, 2026 09:14

Fix static search links on prefixed deployments

1d25468

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: Cursor <cursoragent@cursor.com>

Merge remote-tracking branch 'origin/main' into important-beet

6cf22ef

Co-authored-by: Cursor <cursoragent@cursor.com> # Conflicts: # docs/configure/content-set/index.md

Mpdreamz approved these changes Jul 28, 2026

View reviewed changes

Mpdreamz merged commit 06adbeb into main Jul 28, 2026
24 checks passed

Mpdreamz deleted the important-beet branch July 28, 2026 14:31

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add static search to isolated documentation builds - #3709

Add static search to isolated documentation builds#3709
Mpdreamz merged 14 commits into
mainfrom
important-beet

reakaleek commented Jul 23, 2026 •

edited by Mpdreamz

Loading

Uh oh!

reakaleek commented Jul 28, 2026 •

edited

Loading

Uh oh!

Mpdreamz commented Jul 28, 2026

Uh oh!

Mpdreamz commented Jul 28, 2026

Uh oh!

Mpdreamz commented Jul 28, 2026

Uh oh!

Mpdreamz commented Jul 28, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

reakaleek commented Jul 23, 2026 • edited by Mpdreamz Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why

What

How

Uh oh!

reakaleek commented Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Mpdreamz commented Jul 28, 2026

Uh oh!

Mpdreamz commented Jul 28, 2026

Pagefind.Net 0.4.1 — meta field ordering fix

Uh oh!

Mpdreamz commented Jul 28, 2026

Summary of changes added to this PR

Pagefind.Net 0.4.0 → 0.4.1

Switch to AddHtmlRecord API

Fix CLI pages indexed twice

Uh oh!

Mpdreamz commented Jul 28, 2026

Index size optimization & follow-up

Content limits (this commit)

Impact

What loads when

Follow-up: header load time

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

reakaleek commented Jul 23, 2026 •

edited by Mpdreamz

Loading

reakaleek commented Jul 28, 2026 •

edited

Loading

Switch to `AddHtmlRecord` API